Model for serving exploration traffic

ABSTRACT

One or more computing devices, systems, and/or methods for implementing a model for serving exploration traffic are provided. An amount of spend by a content provider to provide content items of the content provider through a content serving platform to client devices of users is determined. A number of exploration impressions of users viewing exploration content items of the content provider over a timespan is determined. A return on exploration impression metric is determined for the content provider based upon a ratio of the amount of spend to the number of exploration impressions. The return on exploration metric is used to rank available exploration content items of content providers for serving exploration traffic.

BACKGROUND

A content serving platform may be configured to provide content items toclient devices for display to users, such as display through a mobileapplication, a website, or other user interface. For example, thecontent serving platform may provide articles, images, links to content,videos, audio messages, recommendations, and/or a variety of othercontent items to the client devices. In an example, a user on a mobiledevice may access an email application. The email application maydetermine that two content items can be displayed within an email userinterface through which emails of the user are displayed. The emailapplication may transmit a request over a network from the mobile deviceto a computing device, such as a server, hosting the content servingplatform. The content serving platform may utilize a user engagementmodel to rank content items based upon bid values that content providerswill pay if content items are engaged with by users (e.g., a user clicksa content item, the user purchases a service or item described by acontent item, etc.) and predicted likelihoods that users will engagewith the content item, which may be based upon content item attributesof content items and user attributes of user. In this way, the twohighest ranked content items may be selected and transmitted over thenetwork to the mobile device for display through the email userinterface.

SUMMARY

In accordance with the present disclosure, one or more computing devicesand/or methods for implementing a model for serving exploration trafficare provided. A content serving platform is configured to providecontent items to client devices for display to users. A plurality ofcontent providers may submit bids through the content serving platformto bid on opportunities to display content items to users. For example,an opportunity may correspond to a website requesting a content itemfrom the content serving platform to display to a user visiting thewebsite. The content serving platform may rank available content itemsbased upon bid values of bids submitted by content providers of theavailable content items and based upon predictions by a user engagementmodel of predicted likelihoods of the user engaging with each of theavailable content items. The content serving platform may maintain thecontent items as being available for serving non-exploration traffic(e.g., maintained within a non-exploration bucket), which is used by theuser engagement model as a repository of available content items toserve for non-exploration traffic based upon bid values and predictedlikelihoods of user engagement (e.g., 90% of requests for content itemsmay be deemed as non-exploration traffic, and thus are directed to thenon-exploration bucket). The content items are served fornon-exploration traffic in a deterministic manner using the userengagement model to rank content items and select one or more highestranked content items to serve.

When content items become newly available to the content servingplatform, those content items are treated as exploration content itemsbecause the user engagement model has not yet been trained to accuratelydetermine the target audience of users for the content items. Thus, theuser engagement model is unable to create accurate predicted likelihoodsof user engagement for the exploration content items. In an example, theexploration content items are maintained within an exploration bucketthat is separate from the non-exploration bucket. The explorationcontent items within the exploration bucket are served for explorationtraffic (e.g., 10% of requests for content items may be deemed asexploration traffic, and thus are directed to the exploration bucket).User engagement with the exploration content items (e.g., whether a userclicked a content item, whether the user purchased an item based uponviewing a content item, whether the user scrolled past and ignore acontent item, etc.) may be tracked and used to train the user engagementmodel to predict likelihoods of user engagement with the explorationcontent items. Once sufficiently trained or a threshold number ofimpressions have occurred for an exploration content item, theexploration content item may be moved from the exploration bucket to thenon-exploration bucket as a content item that can be served fornon-exploration traffic using the user engagement model. Other variousconditions may trigger the exploration item being moved from theexploration bucket to the non-exploration bucket.

Because exploration content items are served in a non-deterministicmanner without the benefit of the user engagement model (e.g., randomlyselected and served), little benefit may be gained from servingexploration traffic with exploration content items other than fortraining the user engagement model to understand an audience of usersthat would be interested in a particular exploration content item (e.g.,most users may not engage with the exploration content items). Thus,exploration traffic may be “expensive.” Furthermore, certain contentproviders may exploit the content serving platform by obtaining morethan their fair share of exploration content item impressions (e.g.,exploration content items being viewed by users). This can occur when acontent provider defines and submits a large number of explorationcontent items to the content serving platform and/or by making multipleaccounts. This is because exploration content items are served in anon-deterministic manner, and thus a content provider with moreexploration content items will receive a greater probability of theirexploration content items being selected to serve exploration traffic.

Accordingly, as provided herein, exploration content items are rankedusing return on exploration impression metrics for serving explorationtraffic. The return on exploration impression metrics may also be usedto assign percentages of exploration traffic to content providers (e.g.,5% of exploration traffic may be served using content items of a firstcontent provider, 12% of exploration traffic may be served using contentitems of a second content provider, etc.). A return on explorationimpression metric for a content provider is derived from an amount ofspend of the content provider over a timespan (e.g., an amount spent bythe content provider with the content serving platform for servingcontent items of the content provider for non-exploration traffic) and anumber of exploration impressions previously provided for the contentprovider over the timespan (e.g., an amount of views of explorationcontent items over a timespan, such as a number of times a content itemis served for exploration traffic over the timespan). That is, an amountof spend over a timespan by a content provider to provide content itemsof the content provider through the content serving platform to clientdevices of users is determined (e.g., a total amount of bid values paidby the content provider to the content serving platform). The amount ofspend may correspond to spend associated with content items of thecontent provider within the non-exploration bucket and/or may correspondto a total spend of a content provider for all traffic, such asexploration and non-exploration traffic. The number of explorationimpressions of users viewing exploration content items of the contentprovider over the timespan is determined. A return on explorationimpression metric for the content provider is determined based upon theamount of spend divided by the number of exploration impressions. Thereturn on exploration impression metric may be used to rank explorationcontent items to serve for exploration traffic (e.g., a highest rankedexploration content item may be provided to a client device in responseto a request from the client device for a content item to display to auser).

In an embodiment, a percentage of exploration traffic is assigned to thecontent provider based upon the return on exploration impression metriccalculated for the content provider. In an example, the percentage ofexploration traffic is allocated directly based upon the return onexploration impression metric (e.g., the return on explorationimpression metric for the content provider is divided by a sum of returnon exploration impression metrics of all content providers). In anexample, the percentage of exploration traffic is allocated by applyinga function to the return on exploration impression metric (e.g., aranking of return to exploration impression metric and allocating i/npercentage to advertiser i). In an example, the percentage ofexploration traffic is allocated by adjusting a current percentage(e.g., from a previous iteration of identifying return on explorationimpression metrics; an adjustment based upon a comparison to an averagereturn on exploration metric of all content providers, which is notlimited to a comparison to a previous metric of a same content provider;etc.) by a relatively small amount towards a new return on explorationimpression metric (e.g., increasing the previous allocation percentageif the current return on exploration impression metric is larger thanthe previous return on exploration impression metric and decreasing theprevious allocation percentage if the current return on explorationimpression metric is smaller than the previous return on explorationimpression metric).

DESCRIPTION OF THE DRAWINGS

While the techniques presented herein may be embodied in alternativeforms, the particular embodiments illustrated in the drawings are only afew examples that are supplemental of the description provided herein.These embodiments are not to be interpreted in a limiting manner, suchas limiting the claims appended hereto.

FIG. 1 is an illustration of a scenario involving various examples ofnetworks that may connect servers and clients.

FIG. 2 is an illustration of a scenario involving an exampleconfiguration of a server that may utilize and/or implement at least aportion of the techniques presented herein.

FIG. 3 is an illustration of a scenario involving an exampleconfiguration of a client that may utilize and/or implement at least aportion of the techniques presented herein.

FIG. 4 is a flow chart illustrating an example method for implementing amodel for serving exploration traffic.

FIG. 5A is a component block diagram illustrating an example system forimplementing a model for serving exploration traffic.

FIG. 5B is a component block diagram illustrating an example system forimplementing a model for serving exploration traffic, where anexploration model is populated with entries for content providers.

FIG. 5C is a component block diagram illustrating an example system forimplementing a model for serving exploration traffic, where anexploration model is populated with entries for content items.

FIG. 6 is an illustration of a scenario featuring an examplenon-transitory machine readable medium in accordance with one or more ofthe provisions set forth herein.

DETAILED DESCRIPTION

Subject matter will now be described more fully hereinafter withreference to the accompanying drawings, which form a part hereof, andwhich show, by way of illustration, specific example embodiments. Thisdescription is not intended as an extensive or detailed discussion ofknown concepts. Details that are known generally to those of ordinaryskill in the relevant art may have been omitted, or may be handled insummary fashion.

The following subject matter may be embodied in a variety of differentforms, such as methods, devices, components, and/or systems.Accordingly, this subject matter is not intended to be construed aslimited to any example embodiments set forth herein. Rather, exampleembodiments are provided merely to be illustrative. Such embodimentsmay, for example, take the form of hardware, software, firmware or anycombination thereof.

1. Computing Scenario

The following provides a discussion of some types of computing scenariosin which the disclosed subject matter may be utilized and/orimplemented.

1.1. Networking

FIG. 1 is an interaction diagram of a scenario 100 illustrating aservice 102 provided by a set of servers 104 to a set of client devices110 via various types of networks. The servers 104 and/or client devices110 may be capable of transmitting, receiving, processing, and/orstoring many types of signals, such as in memory as physical memorystates.

The servers 104 of the service 102 may be internally connected via alocal area network 106 (LAN), such as a wired network where networkadapters on the respective servers 104 are interconnected via cables(e.g., coaxial and/or fiber optic cabling), and may be connected invarious topologies (e.g., buses, token rings, meshes, and/or trees). Theservers 104 may be interconnected directly, or through one or more othernetworking devices, such as routers, switches, and/or repeaters. Theservers 104 may utilize a variety of physical networking protocols(e.g., Ethernet and/or Fiber Channel) and/or logical networkingprotocols (e.g., variants of an Internet Protocol (IP), a TransmissionControl Protocol (TCP), and/or a User Datagram Protocol (UDP). The localarea network 106 may include, e.g., analog telephone lines, such as atwisted wire pair, a coaxial cable, full or fractional digital linesincluding T1, T2, T3, or T4 type lines, Integrated Services DigitalNetworks (ISDNs), Digital Subscriber Lines (DSLs), wireless linksincluding satellite links, or other communication links or channels,such as may be known to those skilled in the art. The local area network106 may be organized according to one or more network architectures,such as server/client, peer-to-peer, and/or mesh architectures, and/or avariety of roles, such as administrative servers, authenticationservers, security monitor servers, data stores for objects such as filesand databases, business logic servers, time synchronization servers,and/or front-end servers providing a user-facing interface for theservice 102.

Likewise, the local area network 106 may comprise one or moresub-networks, such as may employ differing architectures, may becompliant or compatible with differing protocols and/or may interoperatewithin the local area network 106. Additionally, a variety of local areanetworks 106 may be interconnected; e.g., a router may provide a linkbetween otherwise separate and independent local area networks 106.

In the scenario 100 of FIG. 1, the local area network 106 of the service102 is connected to a wide area network 108 (WAN) that allows theservice 102 to exchange data with other services 102 and/or clientdevices 110. The wide area network 108 may encompass variouscombinations of devices with varying levels of distribution andexposure, such as a public wide-area network (e.g., the Internet) and/ora private network (e.g., a virtual private network (VPN) of adistributed enterprise).

In the scenario 100 of FIG. 1, the service 102 may be accessed via thewide area network 108 by a user 112 of one or more client devices 110,such as a portable media player (e.g., an electronic text reader, anaudio device, or a portable gaming, exercise, or navigation device); aportable communication device (e.g., a camera, a phone, a wearable or atext chatting device); a workstation; and/or a laptop form factorcomputer. The respective client devices 110 may communicate with theservice 102 via various connections to the wide area network 108. As afirst such example, one or more client devices 110 may comprise acellular communicator and may communicate with the service 102 byconnecting to the wide area network 108 via a wireless local areanetwork 106 provided by a cellular provider. As a second such example,one or more client devices 110 may communicate with the service 102 byconnecting to the wide area network 108 via a wireless local areanetwork 106 provided by a location such as the user's home or workplace(e.g., a WiFi (Institute of Electrical and Electronics Engineers (IEEE)Standard 802.11) network or a Bluetooth (IEEE Standard 802.15.1)personal area network). In this manner, the servers 104 and the clientdevices 110 may communicate over various types of networks. Other typesof networks that may be accessed by the servers 104 and/or clientdevices 110 include mass storage, such as network attached storage(NAS), a storage area network (SAN), or other forms of computer ormachine readable media.

1.2. Server Configuration

FIG. 2 presents a schematic architecture diagram 200 of a server 104that may utilize at least a portion of the techniques provided herein.Such a server 104 may vary widely in configuration or capabilities,alone or in conjunction with other servers, in order to provide aservice such as the service 102.

The server 104 may comprise one or more processors 210 that processinstructions. The one or more processors 210 may optionally include aplurality of cores; one or more coprocessors, such as a mathematicscoprocessor or an integrated graphical processing unit (GPU); and/or oneor more layers of local cache memory. The server 104 may comprise memory202 storing various forms of applications, such as an operating system204; one or more server applications 206, such as a hypertext transportprotocol (HTTP) server, a file transfer protocol (FTP) server, or asimple mail transport protocol (SMTP) server; and/or various forms ofdata, such as a database 208 or a file system. The server 104 maycomprise a variety of peripheral components, such as a wired and/orwireless network adapter 214 connectible to a local area network and/orwide area network; one or more storage components 216, such as a harddisk drive, a solid-state storage device (SSD), a flash memory device,and/or a magnetic and/or optical disk reader.

The server 104 may comprise a mainboard featuring one or morecommunication buses 212 that interconnect the processor 210, the memory202, and various peripherals, using a variety of bus technologies, suchas a variant of a serial or parallel AT Attachment (ATA) bus protocol; aUniform Serial Bus (USB) protocol; and/or Small Computer SystemInterface (SCI) bus protocol. In a multibus scenario, a communicationbus 212 may interconnect the server 104 with at least one other server.Other components that may optionally be included with the server 104(though not shown in the schematic architecture diagram 200 of FIG. 2)include a display; a display adapter, such as a graphical processingunit (GPU); input peripherals, such as a keyboard and/or mouse; and aflash memory device that may store a basic input/output system (BIOS)routine that facilitates booting the server 104 to a state of readiness.

The server 104 may operate in various physical enclosures, such as adesktop or tower, and/or may be integrated with a display as an“all-in-one” device. The server 104 may be mounted horizontally and/orin a cabinet or rack, and/or may simply comprise an interconnected setof components. The server 104 may comprise a dedicated and/or sharedpower supply 218 that supplies and/or regulates power for the othercomponents. The server 104 may provide power to and/or receive powerfrom another server and/or other devices. The server 104 may comprise ashared and/or dedicated climate control unit 220 that regulates climateproperties, such as temperature, humidity, and/or airflow. Many suchservers 104 may be configured and/or adapted to utilize at least aportion of the techniques presented herein.

1.3. Client Device Configuration

FIG. 3 presents a schematic architecture diagram 300 of a client device110 whereupon at least a portion of the techniques presented herein maybe implemented. Such a client device 110 may vary widely inconfiguration or capabilities, in order to provide a variety offunctionality to a user such as the user 112. The client device 110 maybe provided in a variety of form factors, such as a desktop or towerworkstation; an “all-in-one” device integrated with a display 308; alaptop, tablet, convertible tablet, or palmtop device; a wearable devicemountable in a headset, eyeglass, earpiece, and/or wristwatch, and/orintegrated with an article of clothing; and/or a component of a piece offurniture, such as a tabletop, and/or of another device, such as avehicle or residence. The client device 110 may serve the user in avariety of roles, such as a workstation, kiosk, media player, gamingdevice, and/or appliance.

The client device 110 may comprise one or more processors 310 thatprocess instructions. The one or more processors 310 may optionallyinclude a plurality of cores; one or more coprocessors, such as amathematics coprocessor or an integrated graphical processing unit(GPU); and/or one or more layers of local cache memory. The clientdevice 110 may comprise memory 301 storing various forms ofapplications, such as an operating system 303; one or more userapplications 302, such as document applications, media applications,file and/or data access applications, communication applications such asweb browsers and/or email clients, utilities, and/or games; and/ordrivers for various peripherals. The client device 110 may comprise avariety of peripheral components, such as a wired and/or wirelessnetwork adapter 306 connectible to a local area network and/or wide areanetwork; one or more output components, such as a display 308 coupledwith a display adapter (optionally including a graphical processing unit(GPU)), a sound adapter coupled with a speaker, and/or a printer; inputdevices for receiving input from the user, such as a keyboard 311, amouse, a microphone, a camera, and/or a touch-sensitive component of thedisplay 308; and/or environmental sensors, such as a global positioningsystem (GPS) receiver 319 that detects the location, velocity, and/oracceleration of the client device 110, a compass, accelerometer, and/orgyroscope that detects a physical orientation of the client device 110.Other components that may optionally be included with the client device110 (though not shown in the schematic architecture diagram 300 of FIG.3) include one or more storage components, such as a hard disk drive, asolid-state storage device (SSD), a flash memory device, and/or amagnetic and/or optical disk reader; and/or a flash memory device thatmay store a basic input/output system (BIOS) routine that facilitatesbooting the client device 110 to a state of readiness; and a climatecontrol unit that regulates climate properties, such as temperature,humidity, and airflow.

The client device 110 may comprise a mainboard featuring one or morecommunication buses 312 that interconnect the processor 310, the memory301, and various peripherals, using a variety of bus technologies, suchas a variant of a serial or parallel AT Attachment (ATA) bus protocol;the Uniform Serial Bus (USB) protocol; and/or the Small Computer SystemInterface (SCI) bus protocol. The client device 110 may comprise adedicated and/or shared power supply 318 that supplies and/or regulatespower for other components, and/or a battery 304 that stores power foruse while the client device 110 is not connected to a power source viathe power supply 318. The client device 110 may provide power to and/orreceive power from other client devices.

2. Presented Techniques

One or more systems and/or techniques for implementing a model forserving exploration traffic are provided. A content serving platform mayspend a substantial amount of storage resources, computing resources,and network bandwidth to store, select, and provide content items toclient devices over a network. Thus, the accuracy of selecting andtransmitting certain content items over the network to client devices ofparticular users that may have an interest in such content items isimportant so that storage resources, computing resources, and networkbandwidth is not wasted by otherwise providing content items to usersthat may ignore the content items because the content items areirrelevant to those users. To improve the accuracy of selecting contentitems that will be engaging and relevant to users, the content servingplatform may utilize a user engagement model for ranking content itemsto provide to users through a bidding process.

In an example of the bidding process, content providers may submit bidsto the content serving platform to bid on an opportunity to provide acontent item to a client device (e.g., to display an image, a video, amessage, text, a link to a website, an article, a recommendation, orother content item through a user interface, a mobile application, awebsite, etc.). A bid by a content provider will have a bid valuecorresponding to an amount the content provider will pay if the userengages with a content item of the content provider in a certain manner(e.g., viewing the content item, clicking the content item, purchasingan item or performing an action after viewing the content item such ascreating an account or signing up to a newsletter, etc.). The userengagement model has been trained to output predicted likelihoods thatparticular users will engage with certain content items based uponcontent item attributes of the content items and user attributes ofusers (e.g., a 23 year old may be more interested in a videogame articlethan a 70 year old grandma). In this way, content items are ranked basedupon their bid values and predicted likelihoods of user engagement sothat content items are served in a deterministic manner in order to moreaccurately provide content items to users that will engage with thosecontent items. In this way, computing resources and network bandwidth isefficiently utilized and not wasted.

Unfortunately, the user engagement model will lack an understandingabout what audience of users will find content items, newly introducedto the content serving platform, interesting and engaging. Accordingly,these newly introduced content items (e.g., content items on which theuser engagement model has not been adequately trained to output accuratepredicted likelihoods of user engagement) are treated as explorationcontent items. Because the exploration content items are provided tousers in a non-deterministic manner without the benefit of the userengagement model, the probability of user engaged is very low (e.g.,content items may be randomly selected to provide to client devices ofusers, whom may ultimate have little interest in such content items).Thus, exploration content items are “costly” because a substantialamount of storage, processing resources, and network bandwidth can bewasted in provided the exploration content items to client devices ofusers that will end up ignoring the exploration content items. Onceenough training data of users engaging or not engaging with anexploration content item has been gathered and used to train the userengagement model to more accurately predict likelihoods of usersinteracting with the exploration content item, the exploration contentitem is treated as a normal content item that is able to utilize theuser engagement model for the bidding process.

Not only are exploration content items “costly,” some content providersmay attempt to exploit the exploration performed by the content servingplatform. Because exploration content items are non-deterministicallyselected (e.g., randomly selected) from an exploration bucket ofavailable exploration content items currently under exploration, acontent provider could define and submit an inordinately large number ofexploration content items to the content serving platform. The moreexploration content items that the content provider has within theexploration bucket, the greater the chance an exploration content itemof the content provider will be selected and provided to a user. Thus,the content provider is exploiting the content serving platform in orderto obtain an unfair share of impressions (e.g., user views) throughexploration traffic. Even if a particular content provider account islimited to having a certain number of simultaneous exploration contentitem actively available through the exploration bucket for servingexploration traffic, the content provider may circumvent this bycreating multiple accounts. Thus, there is a need to more efficientlyand fairly serve exploration traffic with exploration content items in amanner that does not waste storage resources, computing resources, andnetwork bandwidth. It may be appreciated that the exploration bucket maycomprise any data structure, list, designations, or identifiers used todesignate certain content items as being exploration content itemsavailable for serving exploration traffic.

Accordingly, as provided herein, exploration traffic is served in a moreefficient and fair manner to mitigate exploitation. This is achieved byselectively serving exploration traffic with exploration content itemsof content providers based upon ranks assigned to the explorationcontent items using return on exploration impression metrics and/or byselectively serving the exploration traffic based upon percentages ofexploration traffic allocated for the content providers by anexploration model. In particular, an amount of spend by a contentprovider with the content serving platform to provide content items tousers is taken into account when determining a return on explorationimpression metric for the content provider and/or when determining howmuch exploration traffic to allocate to exploration content items of thecontent provider. For example, some content providers may spend asubstantial amount more than other content providers for having thecontent serving platform provide their content items to users. Thus, thespend of content providers is taken into account because contentproviders that typically spend more overall with the content servingplatform should be allocated more exploration traffic than contentproviders that do not spend much with the content serving platform. Thishelps to not reward those content providers that do not ultimately spendmuch on having the content serving platform provide content items tousers. This also helps mitigate instances of exploitation of the contentserving platform where a content provider merely defines a large numberof new content items, which may be used by the content serving platformas exploration content items (e.g., because the new content items arenew to the content serving platform), which can result in exploitationof exploration traffic by the content provider without actually spendingmuch overall.

The number of exploration impressions of users viewing explorationcontent items of the content provider is also taken into account whendetermining the return on exploration impression metric for the contentprovider and/or when determining how much exploration traffic toallocate to exploration content items of the content provider. Thenumber of exploration impressions is taken into account so that acontent provider is not obtaining an unfair share of exploration trafficcompared to other content providers. An exploration impression maycorrespond to a user viewing an exploration content item that was servedfor exploration traffic (e.g., 1% of requests for content items may bedesignated as exploration traffic that is to be served with explorationcontent items, while 99% of requests are designated as non-explorationtraffic that is to be served with content items using a user engagementmodel).

The amount of spend by the content provider over a timespan (e.g., aday, an hour, etc.) and the number of exploration impressions of usersviewing exploration content items of the content provider over thetimespan is used to determine a return on exploration impression (ROEI)metric for the content provider. For example, the return on explorationimpression metric is determined based upon the amount of spend by thecontent provider outside of exploration divided by the number ofexploration impressions of users viewing exploration content items ofthe content provider. The return on exploration impression metriccorresponds to an exploration efficiency of providing the contentprovider with exploration traffic. The more the content provider spendswith the content serving platform and/or the less explorationimpressions consumed by the content provider, the larger the return onexploration impression metric (e.g., users that spend more withouthaving to consume a lot of “costly” exploration traffic are efficientlyutilizing exploration traffic). Thus, exploration content items of thatcontent provider will be ranked relatively higher because of the largerreturn on exploration impression metric assigned to the contentprovider. The less the content provider spends with the content servingplatform and/or the more exploration impressions consumed by the contentprovider, the smaller the return on exploration impression metric (e.g.,users that spend less and consume a substantial amount of “costly”exploration traffic are inefficiently utilizing exploration trafficand/or exploiting exploration by the content serving platform). Thus,exploration content items of that content provider will be rankedrelatively lower because of the smaller return on exploration impressionmetric assigned to the content provider.

Return on exploration impression metrics are determined, for contentproviders that utilize the content serving platform, based upon spend bythe content providers and exploration impressions consumed by thecontent providers during a timespan. The return on explorationimpression metrics are used to rank exploration content items forserving exploration traffic. For example, an exploration content item ofa content provider with a relatively higher return on explorationimpression metric may be ranked higher than an exploration content itemof a content provider with a relatively lower return on explorationimpression metric. In this way, one or more exploration content itemsmay be selected (e.g., a highest ranked exploration content item) toserve a request associated with the exploration traffic (e.g., a requestfrom a client device for a content item).

In an example, percentages of exploration traffic are allocated to eachcontent provider based upon their return on exploration impressionmetrics. In an example, a content provider with a larger return onexploration impression metric, indicating that the content provider isefficiently utilizing exploration traffic, is provided with a largerpercentage of exploration traffic. A content provider with a smallerreturn on exploration impression metric, indicating that the contentprovider is inefficiently utilizing exploration traffic (e.g., wastingcomputing resources, storage resources, and network bandwidth of thecontent serving platform to exploit the content serving platform toprovide exploration content items to users), is provided with a smallerpercentage of exploration traffic.

The exploration model is generated and/or trained to use the percentagesof exploration traffic assigned to content providers for servingexploration content items for exploration traffic. That is, theexploration model is used to select exploration content items to serveexploration traffic (e.g., servicing exploration traffic may correspondto transmitting an exploration content item to a client device inresponse to a request from the client device for one or more contentitems) so that exploration content items of content providers are servedaccording to the percentages of exploration traffic assigned to thecontent providers (e.g., 5% of exploration traffic is served withcontent items of a content provider assigned a 5% percent of explorationtraffic). It may be appreciated that serving traffic (e.g., explorationtraffic and non-exploration traffic) may relate to selecting contentitem(s) or exploration content item(s) to provide to a client device inresponse to a request from the client device for content item(s) todisplay to a user. The exploration model is periodicallyupdated/recomputed (e.g., hourly, daily, weekly, etc.) with currentspend of content providers and current numbers of explorationimpressions. In this way, exploration traffic is efficiently served in adynamic manner in order to avoid exploitation and inefficient resourceutilization of computing devices.

An embodiment of implementing a model for serving exploration traffic inan efficient manner is illustrated by an example method 400 of FIG. 4,and is described in conjunction with FIGS. 5A-5C. A content servingplatform 512 may be hosted by one or more computing devices, such as aserver, a virtual machine, etc. Content providers, such as a firstcontent provider 502, a second content provider 504, a third contentprovider 506, and/or any other number of content providers, may connectto the content serving platform 512 over a network. The content servingplatform 512 may host a bidding process through which content providersmay submit bids of what the content providers will pay for userengagement (e.g., viewing a content item, clicking a content item, orperforming an action after viewing a content item such as purchasing anitem, purchasing a service, signing up for a newsletter, etc.) withcontent items of the content providers (e.g., the first content provider502 may specify a bid value of $0.45 that the first content provider 502will pay if a user clicks on content item, such as an image with a linkto a website).

The content items of the content providers may comprise articles,images, website links, recommendations, text, coupons, videos, and/or awide variety of other types of content. In some embodiments, the contentserving platform 512 may assign content items into one or more buckets(e.g., a bucket may merely be a designation that a content item isavailable for serving one or more types of traffic, such as explorationtraffic and/or non-exploration traffic), while in other embodiments,content items are not assigned to buckets. For example, content itemsfor which a user engagement model 524 is capable of predictinglikelihoods of users engaging with the content items are assigned to anon-exploration bucket 514 for serving non-exploration traffic. Thenon-exploration bucket 514 may comprise any data structure, list,designation, or identifier used to designate certain content items asbeing content items available for serving non-exploration traffic. Apercentage of traffic (e.g., a percentage of requests from clientdevices for content item(s)) is allocated to the non-exploration bucket514, such as 95% of traffic from client devices requesting content itemsfrom the content serving platform 512. In another example, content itemsfor which the user engagement model 524 is not capable of accuratelypredicting likelihoods of users engaging with the content items areassigned to an exploration bucket 516 as exploration content items toexplore. The exploration content items are explored by serving theexploration content items for exploration traffic in order to obtainfeedback information 532 used to better understand what audiences willand will not engage with the exploration content items. The explorationbucket 516 may comprise any data structure, list, designation, oridentifier used to designate certain content items as being explorationcontent items available for serving exploration traffic.

An exploring process is performed for the exploration content itemswithin the exploration bucket 516 in order to learn what audience willengage with such exploration content items (e.g., user attributes ofusers that engaged with an exploration content item can be used tounderstand what types of users will engage with that exploration contentitem). This information may be used to train the user engagement model524 so that the exploration content items may no longer we consideredexploration content items, but can be moved into the non-explorationbucket 514 as content items served for non-exploration traffic using theuser engagement model 524. A percentage of traffic (e.g., a percentageof requests from client devices for content item(s)) is allocated to theexploration bucket 516, such as 5% of traffic from client devicesrequesting content items from the content serving platform 512.

During real-time operation of the content serving platform 512, trafficfrom client devices requesting content items is directed to the variousbuckets maintained by the content serving platform 512, such as 5% oftraffic being directed to the exploration bucket 516 and 95% of trafficbeing directed to the non-exploration bucket 514. The content servingplatform 512 may utilize the user engagement model 524 for rankingcontent items within the non-exploration bucket 514 in order to selectone or more content item(s) 530 to serve in response to a request 528from a client device 510 that is directed to the non-exploration bucket514, such as to display to a user 508 through a website accessed by theclient device 510 or an application being executed by the client device510.

The user engagement model 524 may be generated and/or trained by a userengagement model generator 520. The user engagement model generator 520may train the user engagement model 524 to generate predictedlikelihoods of users engaged with content items based upon attributes ofthe users (e.g., age, gender, location, browsing history, user profiledata, social network data, purchase history, etc.) and content itemattributes of content items (e.g., a topic of a content item, an authorof a content item, a type of content item, etc.). A rank of a contentitem within non-exploration bucket 514 may be determined based upon apredicted likelihood of the user 508 engaging with the content item anda bid value of a bid that a content provider of the content item willpay for user engagement with the content item (e.g., a value paid if theuser clicks on the content item, performs an action after viewing thecontent item, etc.). For example, the rank may be a product of the bidvalue times the predicted likelihood of user engagement. In this way,available content items are ranked, and one or more content item(s) 530are transmitted over a network from the content serving platform 512 tothe client device 510 for display to the user 508 (e.g., highest rankedcontent item(s)).

User engagement or lack thereof for a content item provided to theclient device 510 and/or other content items provided to other clientdevices may be tracked as feedback information 532 by a data center 518.The feedback information 532 may indicate whether the user viewed thecontent item (an impression) or not, whether the user engaged with thecontent item (e.g., did the user click on the content item or perform anaction after viewing the content item), a spend associated with thecontent item (e.g., was the bid value paid because the user engaged withthe content item or did the user not engage with the content item),attributes of the user 508, etc. The data center 518 may provide thefeedback information 532 to the user engagement model generator 520 forfurther training the user engagement model 524.

Exploration content items within the exploration bucket 516 aretypically selected to serve exploration traffic in a non-deterministicmanner without the benefit of the user engagement model 524 that canotherwise provide relatively accurate predicted likelihoods of usersengaging with content items for which the user engagement model 524 hasbeen trained. In an example, when the request 528 for a content item isreceived from the client device 510, the request 528 may be deemed to beexploration traffic and is assigned to the exploration bucket 516 (e.g.,selected to be part of the 5% of overall traffic that is assigned to theexploration bucket 516), an exploration content item may be randomlyselected from the exploration bucket 516 and returned to the clientdevice 510 for display to the user. Unfortunately, thisnon-deterministic manner can be exploited by content providers and isinefficient. Accordingly, as provided herein, an exploration model 526is generated by an exploration model generator 522, and is used to moreefficiently select exploration content items to provide to clientdevices for serving exploration traffic in a manner that mitigatesexploitation and abuse of the content serving platform 512 by contentproviders.

In order to generate the exploration model 526, the exploration modelgenerator 522 takes into account spend data of content providers thatutilize the content serving platform 512, such as the first contentprovider 502, the second content provider 504, and/or the third contentprovider 506. At 402, an amount of spend over a timespan (e.g., an hour,a day, a week, etc.) by the first content provider 502 with the contentserving platform 512 to provide content items of the first contentprovider 502 to client devices of users is determined. In an embodiment,the amount of spend corresponds to bid values paid by the first contentprovider 502 for content items served from the non-exploration bucket514 for non-exploration traffic using the user engagement model 524 toserve the content items of the first content provider 502 in adeterministic manner. In this embodiment, the amount of spend does notinclude spend by the first content provider 502 for exploration contentitems served from the exploration bucket 516 for exploration traffic. Inanother embodiment, the amount of spend may additionally correspond tonot just spend by the first content provider 502 on content items servedfor non-exploration traffic but also spend by the first content provider502 for exploration content items served from the exploration bucket 516for exploration traffic.

Similarly, the exploration model generator 522 determines an amount ofspend by the second content provider 504 over the timespan with thecontent serving platform 512 to provide content items of the secondcontent provider 504 to client devices of users. The exploration modelgenerator 522 determines an amount of spend by the third contentprovider 506 over the timespan with the content serving platform 512 toprovide content items of the third content provider 506 to clientdevices of users. In this way, the exploration model generator 522 willtake into account the different amounts of spend by content providerswhen generating the exploration model 526

At 404, a number of exploration impressions of users viewing explorationcontent items of the first content provider 502 over the timespan isdetermined. For example, an exploration content item of the firstcontent provider 502 may be provided by the content serving platformfrom the exploration bucket 516 to the client device 510 for display tothe user 508 in a non-deterministic manner (e.g., the explorationcontent item may be selected from the exploration bucket 516 withoutusing the user engagement model 524). Feedback information 532 regardingwhether the user 508 viewed the exploration content item may be trackedby the data center 518. The feedback information 532 may indicatewhether the user 508 viewed the exploration content item (an impression)or not. The feedback information 532 may indicate whether there was anyengagement by the user 508 (e.g., whether the user clicked on theexploration content item or performed some action after viewing theexploration content item). The feedback information 532 may compriseuser attributes of the user 508 (e.g., age, gender, location,occupation, etc.), a spend by the first content provider 502 for theimpression or user engagement, etc. The data center 518 may provide thisfeedback information 532 to the exploration model generator 522. In thisway, the exploration model generator 522 may track the number ofexploration impressions of users viewing exploration content items ofthe first content provider 502.

Similarly, the exploration model generator 522 determines a number ofexploration impressions of users viewing exploration content items ofthe second content provider 504 over the timespan. The exploration modelgenerator 522 determines a number of exploration impressions of usersviewing exploration content items of the third content provider 506 overthe timespan. In this way, the exploration model generator 522 will takeinto account the different numbers of exploration impressions of usersviewing exploration content items of different content providers whengenerating the exploration model 526.

At 406, a return on exploration impression metric is determined for thefirst content provider 502 by the exploration model generator 522. Thereturn on exploration impression metric is calculated as a ratio of theamount of spend by the first content provider 502 with the contentserving platform 512 to the number exploration impressions obtained bythe first content provider 502. For example, the return on explorationimpression metric is the amount of spend by the first content provider502 with the content serving platform 512 over the timespan divided bythe number exploration impressions obtained by the first contentprovider 502 over the timespan.

Similarly, the exploration model generator 522 determines a return onexploration impression metric for the second content provider 504 basedupon a ratio of the amount of spend by the second content provider 504with the content serving platform 512 to the number explorationimpressions obtained by the second content provider 504 over thetimespan. The exploration model generator 522 determines a return onexploration impression metric for the third content provider 506 basedupon a ratio of the amount of spend by the third content provider 506with the content serving platform 512 to the number explorationimpressions obtained by the third content provider 506 over thetimespan. In this way, the exploration model generator 522 calculatesreturn on exploration impression metrics for content providers using thecontent serving platform 512.

The exploration model generator 522 outputs the exploration model 526that is derived from return on exploration impression metrics calculatedfor the content providers based upon spend and exploration impressionsduring the timespan. In an example, the exploration model 526 comprisesentries for content providers and/or entries for individual explorationcontent items. For example, a first entry for the first content provider502 may comprise a first identifier of the first content provider 502, areturn on exploration impression metric for the first content provider502 (and/or an exploration percentage of exploration traffic to allocateto the first content provider 502 based upon the return on explorationimpression metric, such as where 0.61% of exploration traffic is to beserved using exploration content items of the first content provider502), and/or other information. For example, the first entry maycomprise a first maximum number of exploration content items of thefirst content provider 502 that can be available at any given point intime for serving exploration traffic (e.g., a maximum number ofexploration content items of the first content provider 502 that can bemaintained within the exploration bucket 516 at any given point intime). The first maximum number of exploration content items may bebased upon the return on exploration impression metric of the firstcontent provider 502 and a configurable base value (e.g., the firstmaximum number of exploration content items may be the configurable basevalue times the percentage of exploration traffic allocated to the firstcontent provider 502). The configurable base value may be determinedbased upon a minimum percentage metric to ensure at least some minimumamount of exploration traffic is served using exploration content itemsof content providers so that any given content provider is not starvedfrom being able to have their exploration content item being provided toclient devices.

The exploration model 526 may comprise a second entry for the secondcontent provider 504. The second entry may comprise a second identifierof the second content provider 504, a return on exploration impressionmetric for the second content provider 504 (and/or an explorationpercentage of exploration traffic to allocate to the second contentprovider 504 based upon the return on exploration impression metric,such as where 0.13% of exploration traffic is to be served usingexploration content items of the second content provider 504), and/orother information. For example, the second entry may comprise a secondmaximum number of exploration content items of the second contentprovider 504 that can be available at any given point in time forserving exploration traffic (e.g., a maximum number of explorationcontent items of the second content provider 504 that can be maintainedwithin the exploration bucket 516 at any given point in time). Theexploration model 526 may comprise a third entry for the third contentprovider 506. The third entry may comprise a third identifier of thethird content provider 506, a return on exploration impression metricfor the third content provider 506 (and/or an exploration percentage ofexploration traffic to allocate to the third content provider 506 basedupon the return on exploration impression metric, such as where 0.42% ofexploration traffic is to be served using exploration content items ofthe third content provider 506), and/or other information. For example,the third entry may comprise a third maximum number of explorationcontent items of the third content provider 506 that can be available atany given point in time for serving exploration traffic (e.g., a maximumnumber of exploration content items of the third content provider 506that can be maintained within the exploration bucket 516 at any givenpoint in time).

The exploration model 526 may comprise entries for particularexploration content items. For example, an entry for an explorationcontent item of a content provider may comprise an identifier of thecontent provider, a content item identifier of the content item, and apercentage of exploration traffic to assign to the content item (e.g.,0.01% of exploration traffic should be served using the explorationcontent item).

At 408, the return on exploration impression metrics of the explorationmodel 526 are used by the content serving platform 512 to rankexploration content items within the exploration bucket 516 for servingexploration traffic. For example, an exploration content item of acontent provider with a relatively higher return on explorationimpression metric may be ranked higher than an exploration content itemof a content provider with a relatively lower return on explorationimpression metric. In this way, one or more exploration content itemsmay be selected (e.g., a highest ranked exploration content item) toserve a request associated with the exploration traffic (e.g., a requestfrom a client device for a content item). In this way, explorationtraffic is efficiently and fairly served using exploration content itemsthat are ranked using the exploration model 526.

In an embodiment, percentages of exploration traffic are assigned to thecontent providers based upon their respective return on explorationimpression metrics. The exploration model generator 522 may include thepercentages of exploration traffic within the exploration model 526 foruse by the content serving platform 512 for selecting and/or rankingexploration content items to serve for exploration traffic directed tothe exploration bucket 516. In an example, a percentage of explorationtraffic is allocated directly based upon a return on explorationimpression metric (e.g., a return on exploration impression metric for acontent provider is divided by a sum of return on exploration impressionmetrics of all content providers). In an example, a percentage ofexploration traffic is allocated by applying a function to a return onexploration impression metric (e.g., a ranking of return to explorationimpression metric and allocating i/n percentage to advertiser i). In anexample, the percentage of exploration traffic is allocated by adjustinga current percentage (e.g., from a previous iteration of identifyingreturn on exploration impression metrics) by a relatively small amounttowards a new return on exploration impression metric (e.g., increasingthe previous allocation percentage if the current return on explorationimpression metric is larger than the previous return on explorationimpression metric and decreasing the previous allocation percentage ifthe current return on exploration impression metric is smaller than theprevious return on exploration impression metric).

In an example, a percentage of exploration traffic may be determinedand/or adjusted based upon a minimum exploration percentage metric. Forexample, if the percentage of exploration traffic (e.g., 0.01%) is lessthan the minimum exploration percentage metric (e.g., 0.03%), then thepercentage of exploration traffic is adjusted up to be the minimumexploration percentage metric. Percentages of exploration traffic forother content providers may be adjusted based upon the percentage ofexploration traffic being adjusted up to be the minimum explorationpercentage metric so that the total percentages of exploration trafficis 100%.

In an embodiment, the exploration model 526 is populated with entriesfor content providers, such as a first entry 550 for a first contentprovider, a second entry 552 for a second content provider, a thirdentry 554 for a third content provider, and/or other entries for othercontent providers, as illustrated by FIG. 5B. The first entry 550 maycomprise an identifier of the first content provider, a percentage ofexploration traffic allocated to the first content provider, and anumber of simultaneous exploration content items that can be maintainedfor the first content provider. The second entry 552 may comprise anidentifier of the second content provider, a percentage of explorationtraffic allocated to the second content provider, and a number ofsimultaneous exploration content items that can be maintained for thesecond content provider. The third entry 554 may comprise an identifierof the third content provider, a percentage of exploration trafficallocated to the third content provider, and a number of simultaneousexploration content items that can be maintained for the third contentprovider.

In an embodiment, the exploration model 526 is populated with entriesfor specific content items, such as a fourth entry 560 for a firstcontent item, a fifth entry 562 for a second content item, a sixth entry564 for a third content item, and/or other entries for other contentitems, as illustrated by FIG. 5C. The fourth entry 560 may comprise anidentifier of the first content provider, a content item identifier ofthe first content item, and a percentage of exploration trafficallocated to the first content item. The fifth entry 562 may comprise anidentifier of the first content provider, a content item identifier ofthe second content item, and a percentage of exploration trafficallocated to the second content item. The sixth entry 564 may comprisean identifier of a fourth content provider, a content item identifier ofthe third content item, and a percentage of exploration trafficallocated to the third content item.

In an embodiment, the exploration model 526 is populated with bothentries for content providers and entries for content items.

User engagement with the exploration content items being served toclient devices of users is tracked as the feedback information 532. Thefeedback information 532 may indicate whether users viewed, clicked,and/or performed actions with respect to the exploration content items.The data center 518 may collect this feedback information 532, andprovide the feedback information 532 to the exploration model generator522. The exploration model generator 522 may periodically (e.g., hourly,daily, weekly, etc.) update/re-compute the exploration model 526 usingthe user engagement data, spend data, and exploration impression data ofthe feedback information 532. For example, new return on explorationimpression metrics may be calculated for a second timespan (subsequentthe timespan for which the return on exploration impression metrics werepreviously calculated) based upon spend by content providers over thesecond timespan and numbers of exploration impressions obtained by thecontent providers during the second timespan. A new exploration model526 may be generated based upon the new return on exploration impressionmetrics for the second timespan. The new exploration model 526 may beused to serve future exploration traffic. User engagement withexploration content items served using the new exploration model 526 maybe tracked and used yet again to update/re-compute the new explorationmodel for subsequent use in serving exploration traffic. In this way,the exploration model 526 may be iteratively updated/re-computed basedupon recent feedback information.

The feedback information 532 is also used by the user engagement modelgenerator 520 to train the user engagement model 524 to generatepredicted likelihoods of users engaging with exploration content items.The user engagement model 524 may output a confidence metric of howconfident the user engagement model 524 is at predicting a likelihood ofusers engaging with an exploration content item. If the confidencemetric is above a threshold, then the exploration content item may beremoved from the exploration bucket 516 and added to the non-explorationbucket 514 as a content item to serve using the user engagement model524 for non-exploration traffic. In another example, the explorationcontent item may be removed from the exploration bucket 516 and added tothe non-exploration bucket 514 as the content item to serve using theuser engagement model 524 for non-exploration traffic based upon athreshold number of impressions occurring for the exploration contentitem. Other various factors may be used to trigger the removal of theexploration content item from the exploration bucket 516 for addition tothe non-exploration bucket 514 as a content item to serve using the userengagement model 524 for non-exploration traffic. In this way,exploration content items and content items may be more efficientlyserved to client devices in a fair manner.

FIG. 6 is an illustration of a scenario 600 involving an examplenon-transitory machine readable medium 602. The non-transitory machinereadable medium 602 may comprise processor-executable instructions 612that when executed by a processor 616 cause performance (e.g., by theprocessor 616) of at least some of the provisions herein. Thenon-transitory machine readable medium 602 may comprise a memorysemiconductor (e.g., a semiconductor utilizing static random accessmemory (SRAM), dynamic random access memory (DRAM), and/or synchronousdynamic random access memory (SDRAM) technologies), a platter of a harddisk drive, a flash memory device, or a magnetic or optical disc (suchas a compact disk (CD), a digital versatile disk (DVD), or floppy disk).The example non-transitory machine readable medium 602 storescomputer-readable data 604 that, when subjected to reading 606 by areader 610 of a device 608 (e.g., a read head of a hard disk drive, or aread operation invoked on a solid-state storage device), express theprocessor-executable instructions 612. In some embodiments, theprocessor-executable instructions 612, when executed cause performanceof operations, such as at least some of the example method 400 of FIG.4, for example. In some embodiments, the processor-executableinstructions 612 are configured to cause implementation of a system,such as at least some of the example system 500 of FIGS. 5A-5C, forexample.

3. Usage of Terms

As used in this application, “component,” “module,” “system”,“interface”, and/or the like are generally intended to refer to acomputer-related entity, either hardware, a combination of hardware andsoftware, software, or software in execution. For example, a componentmay be, but is not limited to being, a process running on a processor, aprocessor, an object, an executable, a thread of execution, a program,and/or a computer. By way of illustration, both an application runningon a controller and the controller can be a component. One or morecomponents may reside within a process and/or thread of execution and acomponent may be localized on one computer and/or distributed betweentwo or more computers.

Unless specified otherwise, “first,” “second,” and/or the like are notintended to imply a temporal aspect, a spatial aspect, an ordering, etc.Rather, such terms are merely used as identifiers, names, etc. forfeatures, elements, items, etc. For example, a first object and a secondobject generally correspond to object A and object B or two different ortwo identical objects or the same object.

Moreover, “example” is used herein to mean serving as an example,instance, illustration, etc., and not necessarily as advantageous. Asused herein, “or” is intended to mean an inclusive “or” rather than anexclusive “or”. In addition, “a” and “an” as used in this applicationare generally be construed to mean “one or more” unless specifiedotherwise or clear from context to be directed to a singular form. Also,at least one of A and B and/or the like generally means A or B or both Aand B. Furthermore, to the extent that “includes”, “having”, “has”,“with”, and/or variants thereof are used in either the detaileddescription or the claims, such terms are intended to be inclusive in amanner similar to the term “comprising”.

Although the subject matter has been described in language specific tostructural features and/or methodological acts, it is to be understoodthat the subject matter defined in the appended claims is notnecessarily limited to the specific features or acts described above.Rather, the specific features and acts described above are disclosed asexample forms of implementing at least some of the claims.

Furthermore, the claimed subject matter may be implemented as a method,apparatus, or article of manufacture using standard programming and/orengineering techniques to produce software, firmware, hardware, or anycombination thereof to control a computer to implement the disclosedsubject matter. The term “article of manufacture” as used herein isintended to encompass a computer program accessible from anycomputer-readable device, carrier, or media. Of course, manymodifications may be made to this configuration without departing fromthe scope or spirit of the claimed subject matter.

Various operations of embodiments are provided herein. In an embodiment,one or more of the operations described may constitute computer readableinstructions stored on one or more computer readable media, which ifexecuted by a computing device, will cause the computing device toperform the operations described. The order in which some or all of theoperations are described should not be construed as to imply that theseoperations are necessarily order dependent. Alternative ordering will beappreciated by one skilled in the art having the benefit of thisdescription. Further, it will be understood that not all operations arenecessarily present in each embodiment provided herein. Also, it will beunderstood that not all operations are necessary in some embodiments.

Also, although the disclosure has been shown and described with respectto one or more implementations, equivalent alterations and modificationswill occur to others skilled in the art based upon a reading andunderstanding of this specification and the annexed drawings. Thedisclosure includes all such modifications and alterations and islimited only by the scope of the following claims. In particular regardto the various functions performed by the above described components(e.g., elements, resources, etc.), the terms used to describe suchcomponents are intended to correspond, unless otherwise indicated, toany component which performs the specified function of the describedcomponent (e.g., that is functionally equivalent), even though notstructurally equivalent to the disclosed structure. In addition, while aparticular feature of the disclosure may have been disclosed withrespect to only one of several implementations, such feature may becombined with one or more other features of the other implementations asmay be desired and advantageous for any given or particular application.

What is claimed is:
 1. A method, comprising: executing, on a processorof a computing device, instructions that cause the computing device toperform operations, the operations comprising: determining an amount ofspend over a timespan by a content provider to provide content items ofthe content provider through a content serving platform to clientdevices of users, wherein the content items of the content provider areavailable for serving non-exploration traffic from client devices in adeterministic manner using a user engagement model; determining a numberof exploration impressions of users viewing exploration content items ofthe content provider over the timespan, wherein the exploration contentitems of the content provider are available for serving explorationtraffic from the client devices in a non-deterministic manner;determining a return on exploration impression (ROEI) metric for thecontent provider based upon a ratio of the amount of spend to the numberof exploration impressions; and utilizing the ROEI metric to rankavailable exploration content items of content providers for serving theexploration traffic.
 2. The method of claim 1, comprising: serving theexploration traffic using exploration content items selected based uponpercentages of exploration traffic assigned to content providers withexploration content items based upon ROEI metrics for the contentproviders.
 3. The method of claim 1, comprising: determining andenforcing a maximum number of simultaneous content items of the contentprovider that can be maintained for serving the exploration trafficbased upon the ROEI and a configurable base value.
 4. The method ofclaim 1, comprising: determining the percentage of exploration trafficfor the content provider based upon a minimum exploration percentagemetric.
 5. The method of claim 1, comprising: generating an explorationmodel for the content provider platform to use for selecting explorationcontent items to serve for the exploration traffic.
 6. The method ofclaim 5, wherein the generating an exploration model comprises:determining ROEI metrics for the content providers; and utilizing theROEI metrics to populate the exploration model with percentages ofexploration traffic to serve using exploration content items of thecontent providers.
 7. The method of claim 5, comprising: populating theexploration model with a first entry for a first content provider,wherein the first entry comprises a first identifier of the firstcontent provider, a first ROEI for the first content provider, and afirst maximum number of simultaneous exploration content items for thefirst content provider.
 8. The method of claim 7, comprising: populatingthe exploration model with a second entry for a second content provider,wherein the second entry comprises a second identifier of the secondcontent provider, a second ROEI for the second content provider, and asecond maximum number of simultaneous exploration content items for thesecond content provider.
 9. The method of claim 1, comprising: trackinguser engagement with an exploration content item for training the userengagement model to predict likelihoods of users engaging with theexploration content item.
 10. The method of claim 9, comprising:removing an exploration content item from an exploration bucket andadding the exploration content item into a non-exploration bucket as acontent item.
 11. The method of claim 1, wherein the determining anamount of spend comprises: determining the amount of spend based upon anamount of non-exploration spend over the timespan by the contentprovider.
 12. The method of claim 1, comprising: populating theexploration model with an entry for an exploration content item.
 13. Acomputing device comprising: a processor; and memory comprisingprocessor-executable instructions that when executed by the processorcause performance of operations, the operations comprising: determiningan amount of spend over a timespan by a content provider to providecontent items of the content provider through a content serving platformto client devices of users, wherein the content items of the contentprovider are used to serve non-exploration traffic; determining a numberof exploration impressions of users viewing exploration content items ofthe content provider over the timespan, wherein the exploration contentitems of the content provider are used to serve exploration traffic;determining a return on exploration impression (ROEI) metric for thecontent provider based upon a ratio of the amount of spend to the numberof exploration impressions; and utilizing the ROEI metric to rankavailable exploration content items of content providers for serving theexploration traffic.
 14. The computing device of claim 13, comprising:generating an exploration model for the content provider platform to usefor selecting exploration content items to serve for explorationtraffic; and periodically updating the exploration model.
 15. Thecomputing device of claim 13, comprising: generating an explorationmodel for the content provider platform to use for selecting explorationcontent items to serve for exploration traffic.
 16. The computing deviceof claim 15, comprising: populating the exploration model with an entryspecifying a first percentage of exploration traffic for a firstexploration content item.
 17. The computing device of claim 13,comprising: adjusting the percentage of exploration traffic based upon aminimum exploration percentage metric.
 18. A non-transitory machinereadable medium having stored thereon processor-executable instructionsthat when executed cause performance of operations, the operationscomprising: determining an amount of spend over a timespan by a contentprovider to provide content items of the content provider through acontent serving platform to client devices of users, wherein the contentitems of the content provider are available for serving non-explorationtraffic; determining a number of exploration impressions of usersviewing exploration content items of the content provider over thetimespan, wherein the exploration content items of the content providerare available for serving exploration traffic; determining a return onexploration impression (ROEI) metric corresponding to a ratio of theamount of spend to the number of exploration impressions; and utilizingthe ROEI metric to rank available exploration content items of contentproviders for serving the exploration traffic.
 19. The non-transitorymachine readable medium of claim 18, wherein the operations comprise:determining and enforcing a maximum number of simultaneous content itemsof the content provider that can be maintained based upon the ROEI and aconfigurable base value.
 20. The non-transitory machine readable mediumof claim 19, wherein the operations comprise: determining theconfigurable base value based upon a minimum percentage metric.